Skip to content

feat(replay): control values + semanticAnnotation on all interactions - #27

Merged
Chinmay-KB merged 11 commits into
feat/replay-coherence-u8-u15from
cursor/control-value-capture-0add
Jul 30, 2026
Merged

feat(replay): control values + semanticAnnotation on all interactions#27
Chinmay-KB merged 11 commits into
feat/replay-coherence-u8-u15from
cursor/control-value-capture-0add

Conversation

@Chinmay-KB

@Chinmay-KB Chinmay-KB commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Per-interaction capture for enrichment:

  1. controlValue for typed standard controls with before/after on settle.
  2. semanticAnnotation on every interaction when Flutter semantics expose an identifier, label, value, or selection flags.

Custom controls

Use Flutter semantics for custom hit targets, or TugboatControlValueScope when the app knows a stable key, typed value, unit, or numeric range.

Semantics(
  button: true,
  value: '30',
  label: 'Duration 30 seconds',
  selected: true,
  child: GestureDetector(...),
)

Privacy

Control values and semantic labels are retained raw. Host apps must treat semantic values as telemetry and avoid placing user PII in them; use TugboatSensitive for content that must remain private.

Release documentation

  • Package/runtime version: 0.4.17
  • Changelog: packages/tugboat/CHANGELOG.md
  • Integration and data-contract details: packages/tugboat/README.md and docs/design/capture-and-fingerprint.md

Verification

  • dart format packages/tugboat/lib packages/tugboat/test
  • dart analyze packages/tugboat
  • flutter test test/control_value_test.dart (25 passing)

@Chinmay-KB Chinmay-KB closed this Jul 29, 2026
@Chinmay-KB Chinmay-KB reopened this Jul 29, 2026
@cursor cursor Bot changed the title feat(replay): capture privacy-safe control values on interactions feat(replay): capture control values + semantic labels on interactions Jul 29, 2026
@cursor cursor Bot changed the title feat(replay): capture control values + semantic labels on interactions feat(replay): control values + semanticAnnotation on all interactions Jul 29, 2026
@Chinmay-KB
Chinmay-KB requested a review from Copilot July 29, 2026 07:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds privacy-safe per-interaction enrichment to Tugboat replay capture by (1) sampling typed “control values” for common controls and (2) attaching a unified semanticAnnotation payload to all interaction events when Flutter semantics expose identifier/label/value/selection flags.

Changes:

  • Capture and serialize controlValue snapshots (including before/after on tap_settled) for standard widgets and semantics-backed targets.
  • Attach semanticAnnotation to tap, tap_settled, swipe, scroll_start, and scroll_end, with ancestor/descendant semantics merged for better labels.
  • Add comprehensive widget tests and update design docs describing payload shapes and privacy rules.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/tugboat/test/control_value_test.dart Adds tests covering control value capture, hashing rules, semantic annotations for taps and scrolls.
packages/tugboat/lib/tugboat.dart Exposes new control/semantics capture APIs and schema version constants via public exports.
packages/tugboat/lib/src/widget_roles.dart Avoids generic callback cast hazards for radio enablement checks by using dynamic access.
packages/tugboat/lib/src/semantics_flags_compat.dart Adds cross-SDK helpers to read enabled/checked/toggled/selected from semantics flags.
packages/tugboat/lib/src/controller.dart Plumbs controlValue and semanticAnnotation into emitted interaction events (tap/settled/swipe/scroll).
packages/tugboat/lib/src/control_value.dart Introduces privacy-safe scalar encoding, TugboatControlValue, and TugboatSemanticAnnotation + merge/build helpers.
packages/tugboat/lib/src/anchors.dart Wires the new control/semantics implementation into the anchors library via a new part file.
packages/tugboat/lib/src/anchor_resolver.dart Adds hit-test + semantics-tree sampling entry points for controlValueAt and semanticAnnotationAt.
docs/design/capture-and-fingerprint.md Documents the new payloads and privacy considerations for semantic tokens.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +26 to +33
// Flutter 3.36+: CheckedState enum (none / isTrue / isFalse / mixed).
try {
if (checked.toString().endsWith('.none')) return null;
if (checked.toString().endsWith('.mixed')) return null;
return checked == CheckedState.isTrue;
} catch (_) {
return null;
}
Chinmay-KB and others added 10 commits July 29, 2026 16:58
fix(replay): align frames with interactions and navigation
feat(replay): consolidate interaction evidence
Record which option or state was chosen for radios, dropdowns, toggles,
sliders, and chips on tap/tap_settled/swipe events. Free-text option
strings are hashed; bools, numbers, enums, and short developer tokens
are retained. Also harden Radio role inspection against typed callbacks.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Extend per-interaction controlValue capture so custom hit targets can
report Flutter semantic value/label tokens when typed widget state is
unavailable. Standard controls still prefer Material/Cupertino state and
merge semantics as supplemental fields under sources.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Attach privacy-safe semantic identifier/label/value to tap, settle,
swipe, and scroll events whenever Flutter semantics expose them. Merge
ancestor/descendant nodes so Material button roles pick up child labels.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

packages/tugboat/lib/src/semantics_flags_compat.dart:31

  • semanticsCheckedFromFlags references CheckedState.isTrue, but this package supports Flutter >=3.35 and CheckedState is only present in newer SDKs, so this will fail to compile on 3.35. Since checked is treated dynamically already, avoid referencing the enum symbol directly and derive the bool from toString() instead.
  try {
    if (checked.toString().endsWith('.none')) return null;
    if (checked.toString().endsWith('.mixed')) return null;
    return checked == CheckedState.isTrue;
  } catch (_) {

packages/tugboat/lib/src/control_value.dart:52

  • When parsing numeric strings, num.tryParse will also accept "NaN"/"Infinity" and produce non-finite doubles, which will later break JSON encoding. Filter those out (or encode them as string tokens) before returning a number kind.
      final asNum = num.tryParse(trimmed);
      if (asNum != null) {
        return TugboatEncodedControlScalar._(kind: 'number', value: asNum);
      }

packages/tugboat/lib/src/control_value.dart:37

  • TugboatEncodedControlScalar.encode returns numbers as-is, which can include non-finite doubles (NaN/Infinity). Those values will cause jsonEncode(...) of session payloads to throw, breaking capture/export. Consider encoding non-finite doubles as string tokens instead.

This issue also appears on line 49 of the same file.

    if (raw is num) {
      return TugboatEncodedControlScalar._(kind: 'number', value: raw);
    }

packages/tugboat/lib/src/controller.dart:2307

  • recordPointerDown calls resolver.buildTapContext(...) and then immediately calls resolver.controlValueAt(position) / resolver.semanticAnnotationAt(position), which each perform their own hit-test walk. This introduces multiple hit tests per tap-down and can be a noticeable overhead in complex trees. Consider extending buildTapContext (or the underlying hit-test loop) to also return controlValue/semanticAnnotation so the tap path is only walked once.
      target = tapContext.target;
      tapInventory = tapContext.inventory;
      controlValue = resolver.controlValueAt(position);
      semantic = resolver.semanticAnnotationAt(position);

@Chinmay-KB
Chinmay-KB force-pushed the cursor/control-value-capture-0add branch from 01d2926 to 03b9ce0 Compare July 30, 2026 05:36
@Chinmay-KB
Chinmay-KB marked this pull request as ready for review July 30, 2026 06:57
@Chinmay-KB
Chinmay-KB merged commit 4373cec into feat/replay-coherence-u8-u15 Jul 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants